ConformalPrediction.jl

Predictive Uncertainty Quantification in Machine Learning

Patrick Altmeyer

Delft University of Technology

July 25, 2023

Talk Agenda

  1. Introduction (5min)
  2. Interactive sprint of ConformalPrediction.jl (10min)
  3. Applications (5min)
  4. Under Construction (5min)
  5. Q&A

Scan this now to follow along … if you’re lucky 😅

QR code redirects to notebook hosted on binder.

Conformal Prediction

Conformal Prediction (CP) works under the premise of turning heuristic measures of Predictive Uncertainty (PU) into rigorous ones through repeated sampling or the use of calibration data.

Example: Split CP

  1. Proper training set and separate calibration set: \(\mathcal{D}_n=\mathcal{D}^{\text{train}} \cup \mathcal{D}^{\text{cali}}\).
  2. Train model on proper training set: \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}(X_i,Y_i)\).
  3. Compute nonconformity scores, \(\mathcal{S}\), using calibration data \(\mathcal{D}^{\text{cali}}\) and fitted model \(\hat\mu_{i \in \mathcal{D}^{\text{train}}}\).
  4. For user-specified coverage ratio \((1-\alpha)\) compute the corresponding quantile, \(\hat{q}\), of \(\mathcal{S}\).
  5. For the given quantile and test sample \(X_{\text{test}}\), form the corresponding conformal prediction set: \(C(X_{\text{test}})=\{y:s(X_{\text{test}},y) \le \hat{q}\}\).

Blog posts

ConformalPrediction.jl

  • Built on top of MLJ.jl.
  • Compatible with Flux.jl through MLJFlux.jl.
  • Many state-of-the-art CP methods implemented for regression, classification and time series modelling.

🏃 Interactive Session

🔍 Applications

Conformal Chatbot

Conformal Intent Classification and Clarification (CICC) won 1st 🏆 at ING Global Experiment Week 2023.

Figure 1: High-level overview of Conformal Intent Classifier.

Figure 2: Demo of a REPL-based conformalized intent classifier.

CICC substantially outperforms baseline approaches (e.g. top-\(K\)).

Blog post

Building a Conformal Chatbot in Julia (blog, TDS)

Conformal Image Classifier

MNIST classifier trained using MLJFlux.jl.

Blog post

How to Conformalize a Deep Image Classifier (blog, TDS, Forem)

Time Series

Ensemble Batch Prediction Intervals (Xu and Xie 2021) contributed by Mojtaba Farmanbar 📣.

Tutorial

How to Conformalize a Time Series Model (docs)

Figure 3: EnbPI for Victoria electricity demand dataset.

🚧 Under Construction

Differentiability

Contribute

Currently working on full conformal training implementation [#62].

Conformal(ised) Bayes

Contribute

LaplaceRedux.jl interfaced to MLJFlux.jl. Planning to add both ideas [#64].

🗨️ and finally …

Trustworthy AI in Julia

Taija collects Julia packages geared towards Trustworthy AI:

Contributions welcome! 😊

References

Angelopoulos, Anastasios N., and Stephen Bates. 2021. “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” https://arxiv.org/abs/2107.07511.
Fong, Edwin, and Chris Holmes. 2021. “Conformal Bayesian Computation.” arXiv. https://doi.org/10.48550/arXiv.2106.06137.
Stutz, David, Krishnamurthy Dj Dvijotham, Ali Taylan Cemgil, and Arnaud Doucet. 2022. “Learning Optimal Conformal Classifiers.” In. https://openreview.net/forum?id=t8O-4LKFVx.
Xu, Chen, and Yao Xie. 2021. “Conformal Prediction Interval for Dynamic Time-Series.” In, 11559–69. PMLR. https://proceedings.mlr.press/v139/xu21h.html.